Information-based Feature Selection

Feature selection is a topic of great interest in applications dealing with high-dimensional datasets. These applications include gene expression array analysis, combinatorial chemistry and text processing of online documents. Using feature selection brings about several advantages. First, it leads to lower computational cost and time. Less memory is needed to store the data and less processing power is needed. Feature selection helps improve the performance of the predictors by avoiding overfitting. It can also capture the underlying connection between the data. And perhaps the most important aspect, it can break through the barrier of high-dimensionality. To select the most relevant subset of features, we need a mathematical tool to measure dependence among random variables. In this work, we use the concept of mutual information. Mutual information is a well-known dependence measure in information theory. For any arbitrary pair of discrete random variables, X ∈ X and Y ∈ Y , Mutual Information is defined as